What & Why of HLD

Low Level Design - Architecture of code

• How to structure the database schema
• The Entities & their Relationships
• The code design patterns
• How are you laying the various components inside your code

LLD helps us keep the code maintainable as the code complexity and team size scales up.

High Level Design - Architecture of the servers / machines that run the code

Interview Question - Staff Engineer @ Google

Given a file containing strings, sort the strings in dictionary (lexicographic) order.

input

cat, dog, apple, laptop, class, high, level, design

output

apple, cat, class, design, dog, high, laptop, level

Very very easy!

with open('data.txt', 'r') as f:

    data = f.readlines()

print(sorted(data))

Catch!

There is 50 Petabytes of data

1 bit: fundamental unit of information

1 nibble: 4 bits

1 byte: 8 bits

1 Kilobyte: 1,000 bytes                                           1 Kibbibyte (1 KiB) = 1,024 bytes

                                                                                (Kilo = 1,000     Kibbi = 2^10 = 1,024)

1 Megabyte: 1 million bytes

1 Gigabyte: 1 billion bytes

1 Terabyte: 1 trillion bytes

1 Petabyte: 1 quadrillion bytes

50 Petabytes: 50,000,000 gigabytes!

Q: Will this amount of data fit in your RAM?

No!

Q: Will it fit in your HDD?

No!

This data must be stored in a distributed manner.

Google has more than 10 million servers across the globe.

Collect data => Sort => Store the data back

Q: What can go wrong?

1. Network Issues

1. slow
2. down

2. Server issues

1. slow
2. down
3. malicious

3. Heterogeneity issues

1. different hardware models
2. different operating systems installed

4. Human error

1. software is buggy

5. Data error

1. hard disk is corrupt

Despite all these challenges you must complete the task, both efficiently, and correctly!

Simple problems become challenging at large scale!

High Level Design is the study of problems that arise at scale, and the solutions for those problems

As we go from 100 users to 1 billion users (planet scale / web scale) what challenges arise, and how we handle them!

Scale

1. Amount of data
2. Number of requests per second

Case Study - del.icio.us  A simple bookmarking service

del.icio.us was launched in 2003, created by Joshua

Youtube: 2004

Amazon AWS: 2006

Google Chrome: 2008

Motivation

https://en.wikipedia.org/wiki/Delicious_(website)

Most people did not have internet at their home.

Any bookmarks that you took on one system are not available on the other systems - because bookmarks are local to the browser

Let's create a bookmarking service - people can store their personal bookmarks on my website.

My users will be able to access their bookmarks from any system, as long as they have an internet connection.

Minimal Viable Product (MVP)

Product: solves a problem

Viable: usable - demonstrates the solution

Minimal: minimal set of features needed to be viable

Proof of concept / Prototype

1. User should be able to identify themselves: registration + login
   note: logout can be implemented later
   
2. User should be able to add bookmarks
   
3. User should be able to view their bookmarks

deletion / updating / ... can be implemented later. Not needed for MVP.

How Internet Works

Internet Service Provider: your phone company that provides you internet

Airtel / Jio / Vi

ACT / Hayai / Starlink / Comcast

IP address: (kinda) unique address assigned to each machine that is connected to a computer network

Q: How does the browser know the IP address associated with a website like google.com?

When Sanjana types the website URL in her browser

1. Joshua must purchase a domain

1. Domain Registrar: intermediary that facilitates domain purchase & manipulation
   Godaddy / Namecheap / Cloudflare /
2. Domain registrar will relay that Joshua has purchased a domain to ICANN
3. ICANN will store the IP address of Joshua's laptop against the domain name that joshua has purchased

Issues with current design

5 Billion of users on the internet

100 billion - 1 trillion devices on the internet

1. ICANN servers will become a bottleneck
2. ICANN servers will become a single point of failure

I Can => I Can't!

Domain Name Service (DNS)

DNS maintains an IP address a list of IP addresses for each domain name

Q: Who maintains the DNS?

ICANN is the central authority.

Actual DNS servers are maintained by other entities.

1. Anyone can create a DNS server
2. Software companies: Google / Amazon / Meta
3. Governments
4. Militaries
5. Research Organizations
6. Educational Institutes
7. Internet Service Providers (ISP): Airtel / BSNL

You can connect to any DNS server!

Q: What if I do not configure any DNS server in my laptop?

Your ISP also has a DNS server - this is configured in your router

Additional Resources

1. Explanation: https://www.cloudflare.com/learning/dns/what-is-dns/
2. Google DNS: https://developers.google.com/speed/public-dns 
3. Deep Dive

• https://network-insight.net/2015/04/21/dns-structure/ 
• https://datatracker.ietf.org/meeting/108/materials/slides-108-tdd-sessb-dns-deep-dive-part-1-pdf-00

4. Configuration: https://www.namecheap.com/support/knowledgebase/article.aspx/319/2237/how-can-i-set-up-an-a-address-record-for-my-domain/ 
5. CName: https://support.dnsimple.com/articles/differences-between-a-cname-alias-url/
6. Domain extensions (.com .us .in .org)

• https://en.wikipedia.org/wiki/List_of_Internet_top-level_domains 
• https://www.godaddy.com/resources/skills/most-common-domain-extensions 

Del.icio.us - Scaling Challenges

Initially there were 100 users. But after some time, this website went viral.

Joshua was getting millions of new bookmarks each day!

Joshua was initially running this website from his personal laptop

System Configuration (back in 2003)

I got my first PC in 2007

• Cost: 35,000 INR

• RAM: 128 MB
• HDD: 40 GB
• CPU: Intel Pentium Code 2 Dual (2 cores, 2.3 Ghz)
• N/W: 8Kbps dialup (bsnl)

In 2003, computers were even worse!

Personal Computer vs Server

• When Joshua goes to sleep, he turns his laptop off - website goes down!
• When Joshua downloads a movie / plays a game - website becomes extremely slow!

Joshua must stop using his laptop for his personal use!

Personal computer is for personal use

Server: Dedicated computer whose sole purpose is to provide a (web) service over the internet

Database Design

users (id, name, password)

user_bookmarks (user_id, URL)

user_id: bigint (8 bytes)

URL: varchar(1000)   (1000 bytes)

https://www.amazon.in/Complete-Balanced-Sardine-Flavour-Pouches/dp/B0BLVJV7VT?pd_rd_w=FoAPt&content-id=amzn1.sym.c6f4280b-256b-4c0a-b056-12ea96d8ae1c&pf_rd_p=c6f4280b-256b-4c0a-b056-12ea96d8ae1c&pf_rd_r=GYW46GCJR692R5YVH0HD&pd_rd_wg=MljSV&pd_rd_r=57e882f4-82cf-4800-80ac-f1bee983ef09&pd_rd_i=B0BLVJV7VT&psc=1&ref_=pd_bap_d_grid_rp_0_1_ec_nped_pr_pd_hp_d_atf_rp_1_t

4 byte integer

• unsigned: 0 ... 2^32    (0 .. 4 billion)
• signed: -2^31 ... (2^31 - 1)     (-2 billion .. 2 billion)

4 byte integer is not large enough to store all the users in the world!

only 4 billion possible values for 8+ billion people!

Q: How large is each row in the user_bookmarks table?

approximately 1 KB

Q: How much data are we adding to the DB each day if we receive 1 million new bookmarks each day?

1 kb / bookmark * 1 million bookmark / day

= 1 kb * 1 million / day

= 10^3 bytes * 10^6 / day

= 10^9 bytes / day

= 1 GB / day

Q: In how many days will we run out of disk space?

40 GB / (1 GB / day)

= 40 days

In 40 days, Joshua's server will run out of disk space!

Q: Is disk space the only resource that needs scaling?

Joshua's website is receiving a ton of requests each day

1. CPU is required: handle the n/w requests, process the db queries
2. Disk is required: storing the data
3. RAM: to handle the request contexts
4. N/W: to handle to requests

Every resource needs to be scaled.

Types of Scaling

1. Vertical Scaling (scaling up)

Replace the existing server with a more powerful system

2. Horizontal Scaling (scaling out)

Add more machines along with existing ones

Alternatively, Joshua could just purchase a lot of cheap laptops

In 1 crore Rs, he can get 1,00,00,000 Rs / 35,000 Rs = 285 laptops!

In fact he will get more! Because Economy of scale. He can purchase directly from the manufacturer.

Maybe he gets 500 laptops

Which scaling technique should we use?

• Vertical scaling is easy: all you have to do is throw money at it
• But vertical scaling has its limitation: you cannot scale infinitely. After a certain point, you have to scale horizontally
• Vertical scaling is limited by the current technology
• Vertical scaling is more costly

• Horizontal scaling is not limited by current technology - you can scale (pretty much) infinitely
• Eventually, you will have to horizontally scale (because vertical scaling will run out)

Horizontal scaling is a huge pain in the ass! It is extremely difficult!

The entirety of the HLD curriculum is about the challenges that arise when scaling horizontally and how to solve them

In reality, you use both. You vertically scale till its financially feasible and simple. When it becomes too costly, or, when you run out of hardware capability, you scale horizontally.

Certain applications (CPU intensive tasks) mandate both vertical & horizontal scaling

1. Delicious: horizontal is all you need. All your systems can be cheap and less powerful
2. Video Processing: you have to scale horizontally, but, each server itself must have a powerful GPU, CPU, lots of RAM. So, you're also scaling vertically
3. Large Language Model (ChatGPT):  7 TB neural network model (you need 7 TB of RAM + 7 TB of GPU memory + 7TB disk). And since you have 500 million users, you also need horizontal scaling

As of Jan 2025, how much can you achieve by vertical scaling?

Typical Server configuration

• CPU: 1 - 16 cores
• RAM: 1 GB - 32 GB
• HDD: 500 GB - 8TB
• N/W: 10 Mbps - 1 Gbps

Max Server configuration (Jan 2025)

• CPU: 370 cores  (x2 on a single motherboard)
• RAM: 12 TB
• HDD: 2 PB
• N/W: 10 TBps

8.45 => 8.55

resume sharp at 8.55 am

class content: 9.30 am  (no new content after 9.30)

doubt session: 10am

Challenges with Horizontal Scaling

Q: Which server's IP should the DNS register?

Load Balancing

Purpose

1. provide a unified view of the entire backend system to the end user
   because the end user doesn't are which exact server handles their request
   
2. distribute the load (requests/data) equally across the other app/db servers

Q: How does the Load Balancer (LB) keep track of the running servers?

Servers can crash at any time!

The load balancer should not forward a user's request to a server which is down

1. Heartbeat: each server periodically pings the LB
   saying "I'm alive!"
   If the LB does not receive heartbeats from a server for a few consecutive times, it will assume that the server is dead
   
2. Healthcheck: LB pings each server periodically
   asks "Are you alive?"
   If the server fails to respond within time (request timeout), then the LB will assume that the server is dead

1. How does LB even know which servers exist?

1. when the server first comes online in the n/w it must ping the LB to register itself
2. you can configure a "range" of IPs in the LB
   10.11.0.1 ... 10.11.2.20

Q: Isn't the Load Balancer a bottleneck?

App server: example - django server to handle bookmarking requests

1. Request go through the OSI layers (webserver will reside at the Application Layer)
2. Deserialize the request to get the payload
3. Authorization - check the user permissions
4. Fetch the resource from DB
5. Process the data
6. Generate a response
7. Serialize the response
8. Send the response

A typical app server can handle 100 - 1000 requests / sec

Load Balancer: does NOT handle the request - it simply forwards it

1. look at the the IP address in the request
2. decide which server to forward the request to
3. re-route the request to that server

A typical LB can easily handle 100,000+ requests / sec

Therefore, LBs do not easily become bottlenecks.

But at the scale of Google, a single LB will not be enough! How to solve that?

Q: What if the Load Balancer crashes? Isn't the LB a Single Point of Failure (SPoF)?

Yes absolutely! If the LB goes down, the entire website goes down!

How do we fix this?

Q: Could we add multiple LBs and have another LB in front of the LBs?

No, that will not work!

We've just added another layer - the problem now happens at the first LB

Solution: we have multiple LBs, and the DNS acts as a LB in front of the LBs

Q: Which server should the request be forwarded to?

Routing algorithm decides that!

1. Round Robin
2. Consistent Hashing

How to succeed in HLD

1. Be curious
   This means asking questions during the doubt session, devouring all the reference material that we provide, exploring interesting topics on your own.
   
2. Think about the internals
   For any app/software/solution that you use, think about the internal design, how they might handle the scale, what unique challenges are they facing, and how does all this effect the features they support (and don't support)
   
3. Think of the business use-case of every feature that you consider.
   For example, how does showing a "blue checkmark for read" help whatsapp?
   
4. Interact with your peers
   HLD is best learn via discussions, by bouncing ideas in a group, by talking about what you've seen at work and why, by talking about the unique tech challenges that you and your company are facing.
   
5. Solve the assignments before the next class
   Don't worry, the assignments are MCQs, so they won't take a long time. They will however require critical thinking, which is exactly what we wish to develop in this module.

Additional Resources

Good resources

1. Reference book: "Designing Data-Intensive Applications: The Big Ideas Behind Reliable, Scalable, and Maintainable Systems" by Martin Kleppmann
2. Alex Xu: https://www.youtube.com/@ByteByteGo 
3. Arpit Bhayani: https://www.youtube.com/@AsliEngineering 
4. Hussein Nasser: https://www.youtube.com/@hnasr 
5. Martin Kleppmann: https://www.youtube.com/@kleppmann 
6. Microservices Reference: https://microservices.io/ 
7. Engineering blogs by companies like Amazon / Meta / Netflix

Resources to avoid

• Medium articles by random authors (go to engineering blogs of companies instead of engineering blogs of people)
• geeksforgeeks
• random youtube videos by bhaiyas & didis